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. ■ Abstract 
C/2 . 

^ , The AgreementMaker system was the leading system in the anatomy 

task of the Ontology Alignment Evaluation Initiative (OAEI) competition 
in 2011. While AgreementMaker did not compete in OAEI 2012, here we 
report on its performance in the 2012 anatomy task, using the same con- 
figurations of AgreementMaker submitted to OAEI 2011. Additionally, 
I we also test AgreementMaker using an updated version of the UBERON 

■ ontology as a mediating ontology, and otherwise identical configurations. 

AgreementMaker achieved an F-measure of 91.8% with the 2011 configu- 
rations, and an F-measure of 92.2% with the updated UBERON ontology. 
Thus, AgreementMaker would have been the second best system had it 
competed in the anatomy task of OAEI 2012, and only 0.1% below the 
F-measure of the best system. 



^ . 1 Introduction 

I AgreementMaker is a powerful, flexible and extensible system for matching on- 

tologies and schemas that has been in development since 2001 [T]. Originally 
focused on geospatial applications, AgreementMaker has since been expanded 
to include many types of matching algorithms and thus handle diverse applica- 
tions. 

In addition to its use in practical applications, the AgreementMaker system has 
been tested in the Ontology Alignment Evaluation Initiative (OAEI) competi- 
tion. The development of AgreementMaker focused in particular on the anatomy 
task, which is a real world application consisting in matching the Adult Mouse 
Anatomy and the NCI Thesaurus describing the human anatomy. In this task, 
AgreementMaker was the best performing system, both in OAEI 2010 and OAEI 
2011. 

While the AgreementMaker system did not compete in the OAEI 2012, here we 



1 



report on its performance in the 2012 anatomy task, using the configurations 
used for OAEI 2011 [2]. AdditionaUy, since one of the matching algorithms 
used in OAEI 2011 uses UBERON [3, as a mediating ontology, we also tested 
AgreementMaker with an updated version of UBERON. 

2 Methods 

The AgreementMaker system was configured as submitted to OAEI 2011 [2]. 
However, in addition to testing AgreementMaker with the UBERON version 
used in OAEI 2011 (release 2011-08-31) we also tested it a more recent version 
of UBERON (release 2012-08-14) [3]. 

We tested the AgreementMaker system on the anatomy task of OAEI 2012, using 
the reference alignment provided by the competition to evaluate its performance. 

3 Results 

The results of the AgreementMaker system in the anatomy task of OAEI 2012 
are presented in Table [TJ As expected, the results obtained with the configura- 
tions submitted to OAEI 2011 match the results obtained in that competition, 
given that no changes were made to the ontologies or reference alignment for the 
task. The results obtained with the updated UBERON ontology had a slightly 
lower precision, but a 0.9% higher recall and thus a higher F-measure. 
In comparison with the systems that participated in OAEI 2012 (Table [2]) 
the recall and F-measure of AgreementMaker are second only to those of the 
GOMMA-bk matcher Furthermore, when using the more recent release of 
UBERON, the AgreementMaker is below GOMMA-bk by only 0.1% F-measure, 
and although its recall is 3.5% lower, its precision is 3.5% higher. 

4 Conclusions 

The results produced by AgreementMaker in the anatomy task indicate that, 
had it competed in OAEI 2012 with the exact same system submitted in the 
previous year, it would have placed second in this task (ranked by F-measure). 
Furthermore, with just an update to the UBERON ontology used as a medi- 
ator, the performance of AgreementMaker would have been only marginally 
below that of the top system, GOMMA-bk. Thus, although the development 
of AgreementMaker did not focus on the anatomy task in the past year, the 
system is still one of the best in this task. 

The future development of the AgreementMaker system should focus particu- 
larly on improving the recall, given that it was significantly below GOMMA-bk 
in this regard. Strategies for this may include the use of additional mediating 
ontologies or other external data sources such as the Wikipedia. 
The development of AgreementMaker should also focus on efficiency, as multiple 
systems ran the anatomy task in under a minute, whereas the running time of 
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Table 1: Results obtained with the AgreementMaker system in the OAEI 2012 anatomy task. 



Matcher 


Runtime 


Size 


Precision 


F-Measure 


Recall 


Recall+ 


Coherent 


AgrMkr2011 


N/A 


1404 


0.954 


0.918 


0.884 


N/A 




AgrMkr2011+ 


N/A 


1421 


0.952 


0.922 


0.893 


N/A 





AgrMkr2011 corresponds to the configuration submitted to OAEI 2011; AgrMkr2011+ is otherwise identical 
but uses the more recent release of the UBERON mediating ontology. 



Table 2: OAEI 2012 anatomy Results. 



Matcher 


Runtime 


Size 


Precision 


F-Measure 


Recall 


Recall+ 


Coherent 


GOMMA-bk 


15 


1534 


0.917 


0.923 


0.928 


0.813 




YAM++ 


69 


1378 


0.943 


0.898 


0.858 


0.635 




CODI 


880 


1297 


0.966 


0.891 


0.827 


0.562 


X 


LogMap 


20 


1392 


0.92 


0.881 


0.845 


0.593 


X 


COMMA 


17 


1264 


0.956 


0.87 


0.797 


0.471 




MapSSS 


453 


1212 


0.935 


0.831 


0.747 


0.337 




WeSeE 


15833 


1266 


0.911 


0.829 


0.761 


0.379 




LogMapLt 


6 


1147 


0.963 


0.829 


0.728 


0.29 




TOAST* 


3464 


1339 


0.854 


0.801 


0.755 


0.401 




ServOMap 


34 


972 


0.996 


0.778 


0.639 


0.054 




ServOMapLt 


23 


976 


0.99 


0.775 


0.637 


0.052 




HotMatch 


672 


989 


0.979 


0.773 


0.639 


0.145 




AROMA 


29 


1205 


0.865 


0.766 


0.687 


0.321 




StringEquiv 




946 


0.997 


0.766 


0.622 







Wmatch 


17130 


1184 


0.864 


0.758 


0.675 


0.157 




Optima 


6460 


1038 


0.854 


0.694 


0.584 


0.133 




Hertuda 


317 


1479 


0.69 


0.681 


0.673 


0.154 




MaasMatch++ 


28890 


2737 


0.434 


0.559 


0.784 


0.501 





These results are presented as published by the OAEI jS]. 
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AgrecmcntMakcr in OAEI 2011 was 10 minutes. 

Finally, the development of AgreementMaker should focus on alignment co- 
herency, which is an aspect under evaluation in the OAEI since 2011, but which 
is currently not addressed by AgreementMaker. Thus, adding a coherence anal- 
ysis module to the AgreementMaker system is paramount. 
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